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Abstract. In this paper we show that the membership problems for 
finitely generated submonoids and for rational subsets are recursively 
equivalent for groups with two or more ends. 



1 Introduction 

Let G be a finitely generated group with finite generating set U. Put S — 
2J U S^ 1 and let 7r: S* — > G be the canonical projection from the free monoid 
on £ to G. The (uniform) submonoid membership problem for G takes as input 
words w, Wi, . . . ,w n £ S* and asks whether tt(w) belongs to the submonoid 
generated by tt(w\), . . . , n(w n ). Note that this problem generalizes the subgroup 
membership problem (or generalized word problem) for G. 

The (uniform) rational subset membership problem for G takes as input 
a word w £ S* and a finite automaton s>4 over S* and asks whether tt(w) G 
-k{L{s^)). Recall that a finite automaton srf over S is a tuple srf = (Q, E, 5, qo, F) 
where SCQxUxQ, q Q ^Q is the initial state and F C Q is the set of final 
states. The language L{srf) recognized by srf is the set of all finite words from 
2J* that label a path from the initial state go to some state from F. A set of the 
form tt(L(m/)) is called a rational subset of G. 

Study of rational subsets of groups began with the pioneering work of Benois 
on free groups [2] and Eilenberg and Schiitzenberger on abclian groups [6]. Ra- 
tional subsets of groups were further studied in [I] , where it was shown that a 
subgroup is a rational subset if and only if it is finitely generated, from which one 
obtains an easy proof of Howson's Theorem (in essence the same as Stallings' 
proof [22]). Recently, there has been renewed interest in the subject of ratio- 
nal subsets of groups [7,8,10,12,13,17,18], especially in connection to solving 
equations over groups [4,5] and the isomorphism problem for toral relatively hy- 
perbolic groups [3]. On the other hand, work of Stephen [23] and Ivanov, Margolis 
and Meakin [9] on the word problem for certain inverse monoids has spurred some 
interest in the submonoid membership problem. In particular, the word problem 
for one-relator inverse monoids motivated the paper [16], where membership in 
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positively generated submonoids of certain groups given by monoid presentations 
is considered. Recently, the authors have shown that the submonoid membership 
problem is undccidablc in free mctabelian groups of rank 2 [14]. 

Trivially, decidability of the rational subset membership problem implies de- 
cidability of the submonoid membership problem. In previous work [13], the 
authors showed that these two problems are recursively equivalent for graph 
groups (also known as right-angled Artin groups or free partially commutative 
groups) and gave the precise class of graph groups for which these problems are 
dccidablc. The authors were also able to prove that these problems are recur- 
sively equivalent for certain amalgamated free products (including free products) 
and HNN extensions with finite edge groups. This led the authors to conjecture 
that the rational subset membership and submonoid membership problems are 
recursively equivalent for groups with two or more ends. The main result of this 
paper is to establish our conjecture. 

Recall that if r is a locally finite graph, then the space of ends of r is the 
projective limit lim7r (r \ F) where F runs over all finite subgraphs of r and 
n (X) is the set of connected components of X. If G is a group with finite 
generating set S , then the number of ends e(G) of G is the cardinality of the 
space of ends of its Cayley graph r with respect to E; it is well known that 
this number depends only on G and not E. Moreover, it is a result of Hopf that 
e(G) is either 0, 1, 2 or is infinite. Clearly e(G) = if and only if G is finite. It is 
well known that e(G) = 2 if and only if G is virtually cyclic. Stallings' famous 
Ends Theorem [20,21] says that e(G) > 2 if and only if G splits non-trivially as 
an amalgamated free product or an HNN extension over a finite subgroup, or, 
cquivalcntly, G has an edge-transitively action without inversions on a simplicial 
tree with no global fixed points and finite edge stabilizers. 

Our main theorem is then the following result. 

Theorem 1. Let G be a finitely generated group with two or more ends. Then 
the submonoid membership and rational subset membership problems for G are 
recursively equivalent. Moreover, there is a fixed rational subset of G with un- 
decidable membership problem if and only if there is a fixed finitely generated 
submonoid of G with undecidable membership problem. 

We remark that the proof of the last statement in Theorem 1 is an exis- 
tence argument: we do not give an algorithm that constructs a finitely generated 
submonoid with undecidable membership problem from a rational subset with 
undecidable membership problem. 

The case that G in Theorem 1 has two ends is clear since a two-ended group is 
virtually cyclic and hence has a decidable rational subset membership problem. 
So the interesting case is a group with infinitely many ends. 

The paper is organized as follows. Section 2 proves some results about groups 
acting on trees that will allow us to reduce Theorem 1 to a special case. This 
special case is then handled in Section 3. 
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2 Groups acting on trees 



The goal of this section is to reduce our considerations to a particular type of 

HNN extension. The notion of a (simplicial) tree is understood in the sense of 

Scrrc [19]. An edge e with initial vertex v and terminal vertex w is written as 

e . . e -1 
v — > w. Then, there is an inverse edge w ► v, which never equals e. The pair 

{e, e" 1 } is called a geometric edge, as it corresponds to an edge of the geometric 

realization of the tree. Often we do not distinguish between an edge and the 

corresponding geometric edge. Let T be a tree. If v,w are vertices of a tree 

T, then [v,w] denotes the geodesic (or reduced) path from v to w in T. Note 

that a path in T is geodesic if and only if it does not contain any backtracking, 

i.e, an edge followed by its inverse edge. An automorphism g of T is without 

inversions if, for each edge e of T, ge ^ e _1 . Let G be a group acting on T (by 

automorphisms of T). Then G acts without inversions on T if every g G G is 

without inversions. Throughout this paper, we tacitly assume that all actions of 

groups on trees are without inversions. We say that G acts edge-transitively on 

T if G acts transitively on the set of geometric edges. The stabilizer in G of a 

vertex or edge x of T will be denoted G x . Of course, G e = G e -i for any edge e. 

A vertex v is called a global fixed point if G = G v . The following lemma is well 

known, but we include a proof for completeness. 

Lemma 2. Let G be a finitely generated group acting edge-transitively on a tree 
T without global fixed points and with finite edge stabilizers (thus G has more 
than one end). Then the kernel of the action of G on T is the unique maximal 
finite normal subgroup of G. 

Proof. Let G e be an edge stabilizer; it is finite by assumption. The kernel X 
of the action of G on T is clearly contained in G e and hence is a finite normal 
subgroup. Let us show that every finite normal subgroup N of G is contained 
in X. Since TV is finite, it fixes some vertex v of T [19, p. 36]. Since G has no 
global fixed point, we can find an element jeG such that gv ^ v. Let [v, gv] be 
the geodesic from v to gv in T. Then since TV = gNg^ 1 fixes v and gv, it fixes 
[v, gv] (since T is a tree) and hence it fixes some edge of this geodesic. Since G 
acts edge-transitively on T and iV is normal, it follows that N fixes every edge 
of T and hence N C X. This completes the proof. □ 

The chief result in this section is a reduction to the case of an HNN extension 
of a very special sort. Recall that if H is a group and ip: A — > B is a partial 
automorphism of H, i.e., an isomorphism between subgroups A and B of H, then 
the corresponding HNN extension is the group * v H given by the presentation 
(H,t | t- 1 at = ip(a) (a e A)). 

We first need some results concerning groups acting on trees, which may be 
of interest in their own right. We recall a result of Tits (cf. [19, p. 63]) that an 
automorphism g of a tree T (without inversions) either fixes a vertex, in which 
case it is called elliptic, or it leaves invariant a unique line T g , called its axis, 
on which it acts via translation by some positive integer length. Elements of 
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the latter sort are called hyperbolic. Of course, hyperbolic elements have infinite 
order. 

Our next lemma is a geometric version of a result proved combinatorially 
(and separately) for amalgamated free products and HNN extensions with finite 
edge groups [11]. 

Lemma 3. Let G be a finitely generated group acting edge-transitively on a 
tree T without global fixed points and with finite edge stabilizers (thus G has 
more than one end). Let v ^ w be an edge of T and suppose that G e is finite 
proper subgroup of both G v and G w . Then there is an element s£G such that 
G e fl sG e s _1 = X where X is the largest finite normal subgroup of G. 

Proof. Since the pointwise stabilizer of a geodesic is the intersection of the stabi- 
lizers of all its edges (and hence has size bounded by \G e \), it follows that there 
is a non-empty finite geodesic path p whose pointwise stabilizer H has smallest 
cardinality amongst all geodesies in T. Without loss of generality, we may as- 
sume that p starts with either e or e _1 (else translate it to do so). Moreover, 
if p starts with e" 1 we can replace e by e _1 and hence we may assume that p 
starts with e. Let (se) e with s e G and e — ±1 be the last edge traversed by 
p. We claim that we can choose p so that e = 1. If this is not the case, i.e., 
the last vertex of p is sv, then we can choose g £ G sv \ G se and consider the 
path q = p(gse). We then have the situation in Figure 1. Any element of G that 



[w, sw\ 



gsw 



Fig. 1. When sv is the last vertex of p. 



fixes q must then fix p and hence the pointwise stabilizer of q coincides with 
H by minimality. Thus replacing s by gs, we are in the desired situation. In 
particular, it follows that we may take s to be a hyperbolic element with [v, sv] 
a fundamental domain for the action of s on its axis T s , as the edges e, se are 
coherent in the sense of [19, p. 62]. Since every element that stabilizes e and se 
stabilizes p, we have H = G e l~l sG e s _1 . We will show that H = X, where X 
is the largest finite normal subgroup of G. Recall from Lemma 2 that X is the 
kernel of the action of G on T. Hence, X C H. It remains to show that H C X, 
i.e., that H stabilizes every edge of T. 

First we show that H is the pointwise stabilizer of T s . Clearly, the pointwise 
stabilizer of T s is contained in H, since p is contained in T s . If the pointwise 
stabilizer of T s is not equal to H , then some edge / of T 8 is not stabilized by some 
element of H. By extending p to a reduced path in the line T s containing /, we 
obtain a reduced path with a smaller pointwise stabilizer than p, a contradiction. 
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Next, observe that H is closed under conjugation by all powers of s since T s 
is (s)-invariant and H is the pointwise stabilizer of T s . Suppose now that ge is 
an edge oiT\T s with g 6 G. Let r be the geodesic path from ge to T s . Clearly, 
H stabilizes ge if and only if it stabilizes s n ge for some n since s n Hs~ n = 
H . Moreover, the geodesic path from s n ge to T s is s n r. Thus without loss of 
generality, we may assume that r ends at a vertex u of s m [v, sv] with m > 2. 
Then p[sw, u]r~ 1 (ge) e is a reduced path (see Figure 2) containing p as an initial 
segment, for an appropriate choice of e = ±1, and hence has pointwise stabilizer 
contained in H. But by choice ofp, the pointwise stabilizer of p[sw, u]r~ 1 (ge) £ 
cannot be properly contained in H. Thus H stabilizes ge, i.e, H stabilizes every 
edge of T and hence H = X. This completes the proof. □ 



(geY 



p 

, A > 

V »- w sv sw 



Fig. 2. The path p[sw, u]r 1 {ge) E . 



Our next proposition uses the element s constructed above to create sub- 
groups of a special form. 

Proposition 4. Let G be a group acting on a tree T . Let v w be an edge of 
T and put N = f] geG gGeg^ 1 . Suppose that: 

(a) [G v : G e ] > 3; 

(b) [G w : G e ] > 2; 

(c) There is an element s E G with sG e s _1 Pi G e = N. 

Then G contains a subgroup isomorphic to an amalgamated free product of the 
form G v *n {N x Z). Moreover, if G e is finite, then G contains a subgroup 
isomorphic to G v *n (N x Z) = H< ljv G v , where In is the identity map on N. 

Proof. We begin by observing that N consists of those element of G that stabilize 
all translates of e. Our third hypothesis (c) then says that there is a translate 
se so that the subgroup of elements fixing both e and se is exactly N. Our 
initial goal is to show that we may assume that the geodesic [v, sv] from v to sv 
contains both e and se. First suppose the geodesic contains exactly one of these 
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edges. Replacing e by se and s by s _1 if necessary, we may assume without 
loss of generality that it contains e. Since G sw ~D G se by (b), we can choose 
g G G sw \ G se . Then we have the situation in Figure 3. Any clement of G that 



[w, sv] 



gse 



gsv 



Fig. 3. When only e is in [v, sv]. 



fixes e and gse must then fix e and se and so gsG e s~ 1 g~ 1 C\G e = N. Moreover, e 
and gse belong to [v, gsv]. Thus replacing s by gs, we are in the desired situation. 

Next suppose that neither e nor se belong to [v, sv]. Choose g G G sw \ G se . 
Then we have the picture in Figure 4. Again any element of G which fixes e 



gse 



gsv 



Fig. 4. When neither e nor se is in [v, sv]. 



and gse must fix also se and hence belong to N. Thus gsG e s~ 1 g~ 1 f)G e — N. 
Moreover, gse is an edge of [i;,<?si>] and so replacing s by gs leads us to the 
previous case where exactly one of the edges is on the geodesic. 

Thus we may now assume that e and se belong to [v, sv]. Choose g G G v \G e . 
Then we have the situation in Figure 5. The edges e and sge are coherent in the 
sense of [19, p. 62]. Thus x — sg is a hyperbolic element with axis T x = (x)[v, sv] 
and [v, sv] is a fundamental domain for the action of x on T x (cf. [19, Sec. 1.6]). 
Let Xi be the connected component of T \ {se} containing sv and let X2 be the 
connected component of T \ {se} containing sw. 

Since [G v : G e ] > 3 by (a) and T x contains only one other (geometric) edge 
incident on v besides e, we can find h G G v \G e so that he ^ T x . Then y = hxh^ 1 
is a hyperbolic element with axis T y — hT x . The axes T y and T x intersect at the 
vertex v — hv, but do not coincide since he G T y \T x . Thus T y n T x is either a 
point, a segment or an infinite ray. Notice that X\ (~)T X and X2 C\T X each contain 
exactly one of the two ends of T x . Interchanging e with se and s with in 
the statement of the proposition if necessary, we may assume that T y n T x does 
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Fig. 5. A part of the axis of x — sg 



not contain the end of T x determined by X^ H T x . Then, since x acts on T x by 
translations in the direction v to sv, by choosing n > large enough, we can 
guarantee that x n T y n T x is contained in X\ and does not contain the vertex sv. 
Putting u = x n yx~ n yields a hyperbolic element u 6 G with axis T u = x n T y 
such that T u fl T x ^ is contained in Xl and does not contain sv. The picture 
is either as in Figure 6 or 7. 




Fig. 6. The axes T x and T u have bounded intersection. 



Let Hi = (N, u) and H2 = G v . Since u is hyperbolic and elements of N 
are elliptic, it follows that (u) fl N = {1}. Consequently, i?i = TV x Z via the 
conjugation action of u on N. Also, if AT is finite, then we can find a power n > 
so that u n acts trivially on N by conjugation. Then u n is a hyperbolic element 
with the same axis as u, but a larger translation length. Replacing u by u n , we 
may then assume that u commutes elementwise with A^ and so Hi = N x Z. 
Also note that Hi n i?2 = TV- Indeed, if /i G Hi, we can write h = u n a with 
a e A 7 ". Then since a £ G v , we have /i e G„ if and only if u n e G„, if and only if 
n = since u is hyperbolic. 

We claim that Hi and i?2 generate a subgroup isomorphic to Hi * jv ^2 • To 
prove this, we use the so-called Ping Pong Lemma [15]. The Ping Pong Lemma 
has the following setup: a group G acting on a set X, subgroups Hi and Hi of 
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Fig. 7. The axes T x and T u have unbounded intersection. 



G with [Hx : N] > 3, [H 2 : N]>2, where N = H x n i? 2 , and non-empty subsets 
Ii,I 2 CI with X 2 ^ Xi such that: 

- hxX 2 C Xi for all hi e #1 \ JV; 

- /i 2 Xi C X 2 for all h 2 eH 2 \ N. 

The conclusion of the lemma is that the subgroup of G generated by i?i and i? 2 
is isomorphic to the amalgamated free product Hx *jv H 2 . We apply the Ping 
Pong Lemma to our subgroups Hi,H 2 . We take X to be T, whereas X\ and 
X 2 have already been defined above. One should think of the edge se as the 
net of the ping pong table X. Note that in our case \H\ : N] — 00, whereas 
\H 2 : N] > [G v : G e ] > 3. 

Let k = u m a g Hi \ N with a e iV. Then a fixes [v, sw] because it fixes e 
and se. Let g be the geodesic from sv to T u r\T x ; it is non-empty by choice of 
u. As any non-trivial element u m of (u) acts as a translation on the axis T u , it 
follows that the geodesic p = [ksv, sv] — [u m sv, sv] must pass through a vertex 
of T u n T x as per Figure 8. Since T u n is contained in Xi, it follows that p is 



X 2 



Xi 



ksv= u m sv 



/ u m q 



Fig. 8. The geodesic p = [fcsv, sw] = [u m sv, sv] 
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contained in X\. Let v G X 2 . Wc claim that k[vo, sv] U [ksv, sv] = [kvo, ksv] Up 
is the geodesic from kv to sv, which implies that this path (and hence kvo) 
is contained in X\ since p is contained in X\. To show that k[v$, sv \ U [fcsi>, sv] 
is a geodesic, note that k[vo,sv] ends with the edge (kse) -1 . Hence, it suffices 
to show that p does not begin with the edge kse. The geodesic p = [u m sv, sv] 
begins with the path u m q, see Figure 8. Since gCIj does not contain se, u m q 
does not contain kse — u m se. We have thus shown kX 2 C Xi. 

On the other hand, if fc G H 2 \ N, then k stabilizes v but not se (since if it 
stabilizes v and se, then it must also stabilize e and so belong to N by assumption 
on s). Since fcse ^ se and few = v, there is a last vertex v on the geodesic [v, sv] 
fixed by fc and vq 7^ st>. Let t>i G Xi. Then by definition of X\, it follows that 
[«i,wo] = U [sv,vo] and hence [kvi,v n ] = k[vi,vo] — k{v\,sv] Uk[sv,v n ]. 

Notice that k[sv,Vo] — [ksv,v ] cannot contain any edge of [v,sv] by choice of 
vq. Therefore, fc[«i,st>] U k[sv,vo] U [fo,sw] = [fcui,sw] (see Figure 9) and hence 
[kvi,sv] contains the edge se. It follows that kv\ G X 2 and so kX\ C X 2 . The 
Ping Pong Lemma now yields that (Hi,H 2 ) = Hi *n H 2 , completing the proof. 

□ 




Fig. 9. The inclusion kXi C X 2 . 



The proposition admits the following corollary. 

Corollary 5. Let G be a finitely generated group splitting non-trivially over a 
finite subgroup A. Suppose that H is an infinite vertex group of the corresponding 
splitting. Then G has a subgroup isomorphic to the HNN extension * lx H where 
X is the largest finite normal subgroup of G and lx ■ X — > X is the identity 
mapping. 

Proof. Let T be the Bass-Scrre tree associated to the splitting of G over A. 
It follows that there is an edge v —* w of T so that G e — A and G v = H. 
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Moreover, since the splitting is non-trivial and also G cannot be an ascending 
HNN extension of H as H is infinite and A is finite, we must have G e C G w . The 
action of G on T is edge-transitively and has no global fixed point. Lemma 3 
provides s £ S with 

sGeS^ 1 n G e = sAs" 1 H A = X. 

Observing that X — f] g( - G gGeg^ 1 , Proposition 4 then allows us to conclude 
that G contains a subgroup isomorphic to H *x (X x Z), which is easily seen to 
be isomorphic to the desired HNN extension. □ 

The following technical result, which is the heart of this paper, will be proved 
in the next section. 

Lemma 6. Let G = * lx H be an HNN extension with X a finite proper nor- 
mal subgroup of H. Then the rational subset membership problem for H can be 
reduced to the submonoid membership problem for G. Moreover, if H has a fixed 
rational subset with undecidable membership problem, then G has a fixed finitely 
generated submonoid with undecidable membership problem. 

Let us now prove Theorem 1 assuming Lemma 6. 



Proof of Theorem 1 For the first statement of Theorem 1, the non-trivial 
direction is to show that the rational subset membership problem for G reduces 
to the submonoid membership problem for G. By Stallings' Ends Theorem [20, 
21] the group G splits non-trivially over a finite subgroup A. By the results of [10, 
12], the rational subset membership problem for G reduces to the rational subset 
membership problem for the vertex group(s) of the splitting. Hence, it suffices to 
show that if H is a vertex group of the splitting, then H has decidablc rational 
subset membership problem. If H is finite, there is nothing to prove, so we may 
assume that H is infinite. Then by Corollary 5, G contains a subgroup isomorphic 
to *i x H. Clearly, also *i x H has a decidable submonoid membership problem. 
The result now follows from Lemma 6. 

The non-trivial direction of the second statement is to show that if G has a 
fixed rational subset with undecidable membership problem, then it has a fixed 
finitely generated submonoid with undecidable membership problem. In [12] the 
membership problem for a fixed rational subset of G is reduced to the member- 
ship problem of a finite number of fixed rational subsets of the vertex groups 
(in the splitting over A) . Hence one of the vertex groups H has a fixed rational 
subset with undecidable membership problem; necessarily H is infinite. Corol- 
lary 5 then applies to allow us to deduce that G has a fixed subgroup of the 
form *i x H where X is a finite proper normal subgroup of H and H has a fixed 
rational subset with undecidable membership problem. Thus by Lemma 6, we 
conclude G has a fixed finitely generated submonoid with undecidable member- 
ship problem. This completes the proof. □ 
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3 The Proof of Lemma 6 



Let us fix an HNN-extension G = (H,t | t~ 1 xt — x (x G X)), where X is a 
proper finite normal subgroup of the finitely generated group H. We assume 
furthermore that H is infinite. The group H acts on X by conjugation and since 
X is finite, the kernel N of this action has finite index in H (and hence is finitely 
generated). In particular, TV is infinite and so N n X is a proper subgroup of 
N . Since decidability of rational subset membership is a virtual property [8], to 
show that H has decidable rational subset membership, it suffices to prove that 
N has decidable rational subset membership. 

Let K — (N, t) < G. Then K centralizes X and so in particular, K n X is 
contained in the center of K. Let us fix a finite group generating set E for N 
and denote by 7r : E* — + N the canonical projection. Without loss of generality 
assume that NDX C E. Suppose that si = (Q, E, S, qo, F) is a finite automaton, 
For a state q G Q, let X(q, si) be the set of all elements from X n N represented 
by a word labeling a loop at state q. Note that these elements form a submonoid 
of XnN and therefore (since XnN is finite) a subgroup of XnN. Let X(si) C E 
be the set [J qe QX(q, s/). 

The proof of the following lemma is quite similar to the proof of Lemma 11 
and Theorem 7 in [13]. 

Lemma 7. From a given finite automaton s/ over E and an element h G N 
(given as a word over E) we can construct effectively a finite subset A C K and 
an element g G K such that h G w(X(s/)*L(s/)) if and only if g G A* . 

Proof. Let si = (Q,E,5,q n7 F). Without loss of generality assume that QCN 
and q > 1 for all q G Q. By introducing e-transitions, we may also assume that 
the set of final states F consists of a single state qf. We will construct a finite 
subset A C K and an clement g G K such that h G ir(X(s/)* L(sf)) if and only 
if 9 G 

Fix an element k G N\X (hence also fc _1 G N\X). Without loss of generality 
we may assume that k G E. For every q G Q C N, define [q] G K by 

[g] = ^fcr 9 . 

Note that [g]x = x[g] for all x G X since K centralizes X. Let 

A = {\q\c\p]- 1 | ( g ,c,p) G 6} and 5 - foblM*?/] -1 . (1) 

where we abuse notation by treating Z 1 as if it were a subset of N. Observe that 
in (1), we have c G £U{1] C N, since we introduced e-transitions. Also A C K. 
Note that X(^/)* is contained in A*. Indeed, if x G X(q, s/), then we can write 
x = a\ ■ ■ ■ a m where there are transitions 

(q,a 1 ,q- L ),...,(q m - 1 ,a m ,q) G S. 

Then, since x commutes with [q], we have 

x = \q\x\q}- 1 = [^a^gij^^qijaafe] -1 • • • [g m -i]a m [g] _1 G A*. 
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It follows that X(j2/)* C A*. 

We claim that h G n(X(^)*L(j^))jf and only if g G Z\* . Let r = E U {i}. 
Let us define a X -cycle to be word in J 1 * of the form 

such that £ > 1, qi, . . . ,qt G Q, hi, . . . ,hf G Z 1 *, and hi ■ ■ ■ hi represents an 
clement from the subgroup X. Since all elements from X commute with all [q] 
(q G Q), an A-cycle equals an element from X in G (actually an element of 

Nnx). 

A word in _T* of the form 

' ' ' [«m]/lm[Pm] _ \ 

where qi,pi, ■ ■ ■ , q m ,Pm G Q and hi, . . . ,h m G Z 1 *, is called X -cycle-free if it 
does not contain an X-cycle as a factor. 

Claim 1. Suppose that pi, . . . ,p m , qi, ■ ■ ■ , q m G Q with Pi ^ qi (1 < i < m) and 
Qi Pi+i (1 < i < m). Then the element 

g = \pi\- 1 [qM- 1 [q2\---\pm]- 1 [q m ] (2) 

has a reduced expression in the HNN extension G starting with t and ending 
with t' 1 . 

Proof of Claim 1. We have 

m m 

l[[p t }- 1 h} = l[^k-H-PH^kt-^ 

i=l i=l 
m 

i=l 

=tpi ( n k-H q '-^kt p ^ i - q ' \k-H q ™- v ™kt- qm . 

The latter word is a reduced word for g of the required form. This establishes 
Claim 1. 

Let Y be the set of elements of the form (2). It follows immediately from the 
claim that any element g' of the form /ioJ/i^i2/2^2 ■ • - yrh r with yi, . . . ,y r G Y , 
hi, ... , h r -i G N \ X, and ho, h r G (N \ X) U {1} has a reduced expression for 
the HNN extension G containing t and t~ x and hence does not belong to N. 
Such elements g' will be called good. I.e., </ is good if it can be written as an 
alternating word in elements of Y and N \ X with at least one factor from Y. 

Claim 2. Assume that in the HNN-extension G 

[n]/i[r] _1 = [(7i]/ii[pi] _1 • • • [q m ]h m \Pm]~ 1 (3) 

where n,r,qi,pi, . . .,q m ,Pm G Q, h,hi, . . . , h m G E* . If IlIlifelMK] -1 is X- 
cycle-free, then either m = 0, n = r, and h = 1, or all of the following hold: 
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— m > 1 and pi = q i+ i for all 1 < i < m, 

— qi = n and p m = r, and 

— h = hi ■ ■ ■ h m in N. 



Proof of Claim 2. We prove Claim 2 by induction over m. If m = 0, then (3) 
becomes 

h=[n]- 1 [r]. 

Since h G N, Claim 1 immediately yields n = r, and hence h = 1, as required. 
Now assume that m > 1. 

Case 1. m>2 and there is 1 < £ < m such that pg = qe+i- Then (3) implies 

(£— 1 \ / m 

niqi}h i \p i ]- 1 )[q i ](hehe +1 )\p e+1 ]- 1 ( J[ 

in G. Since n™i fel^M -1 from (3) is X-cycle-free, also the right hand side of 
this equality is X-cycle-free. Hence, we can apply the induction hypothesis and 
obtain: 

— Pi = Qi+i for all 1 < « < ^ — 1 and all i + 1 < i < to, 

— q\ = n and p m = r, and 

— h = h\ ■ ■ ■ h m in N. 

Since pg = qi+i we obtain pi = qi+\ for all 1 < i < to. 

Case 2. pi ^ q-i+i for all 1 < i < to. We have to show that m = 1, q\ = n, 
p m — r, and h = h\ in N. Choose a set T of coset representatives for N/{NC\X) 
with 1 6 T and write hi = Xih\ with h\ G T and Xi G iVnX, for 1 < i < m. From 
this definition we get h\hi ■ ■ ■ h m — xh^h^ • ■ ■ h' m where x = x\ ■ ■ ■ x m G N n X 
(since N centralizes X). Moreover, we also have 

m m 

W^^r 1 = xW^h'M- 1 

i=l i=l 

since iV n X is central in K. 

Thus, again using that x is central in if, we have 

x- 1 h=[n]- 1 (R[q i ]h>\p i ]-^[r]. (4) 

Since the right hand side of (3) is X-cycle-free, if h\ = 1 (i.e., hi G X), then 
% 7^ Pi, for all 1 < i < m (else [<7i]/ii[pi] _1 is an X-cycle). Also we are assuming 
Pi Qi+i- If K 1) then h'i S N\X. Therefore, if both n ^ q\ and r ^ p m , then 
the right hand side of (4) represents a good element of G and so cannot belong 
to N, as was observed just before the statement of Claim 2. This contradicts 
x^ 1 h G N. Similarly, suppose n = q\ and p m ^ r. Then 



x- 1 h = h' 1 \p 1 }- 1 [l[[ qi ]h' i \p i 



\i=2 
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which is again good, a contradiction. The case n ^ gi and p m = r, is handled 
analogously. If n = qi,p m = r and m > 2, then 

x-'h = hlfa]- 1 (j[ [q^h'M-^ [lm]h' m 

and so is again good, a contradiction. The only remaining case is when n = pi, 
f = Pm and m = 1. Then [n]/i[r] _1 ~ [n]/ii[r] _1 in G and so h = hi in N, as 
required. This proves Claim 2. An immediate consequence of Claim 2 is: 

Claim 3. Assume that in the HNN-extension G 

where (qi,a,i,pi) £ S (i.e., [g^a^Pi] -1 £ Z\) for 1 < i < m. If J]™ i feHbi]" 1 is 
A-cycle-free, then h £ ir(L(g/)). 

The case m > 1 follows immediately from Claim 2. In case m = 0, Claim 2 implies 
q = qj and h — 1. But g = <Z/ implies e 6 and hence /i £ tt(L(£/)). 

Now we are ready to prove that h £ 7r(X( 1 e/)*L( I !y)) if and only if g = [goj/^g/] -1 e 
Z\*. First assume that h £ 7r(A(«e/)*L(^/)). Let h = xa\ ■ ■ ■ a m in N, where 
ai ■ • ■ a m £ L(m/) and x £ A(«e/)*. Hence, there are states go , . . . , q m with 
(qi-i 7 ai 7 qi) £ 5 for 1 < i < m, and g m = g/. It was observed earlier that 
X(sf)* C Moreover, since x commutes with [q ], we obtain in G: 

m 

g = [(7o]/i[<?/] _1 = aj[<?o]aia 2 • • -a^g/] -1 = xjjfe-ijaife] -1 £ A*. 

Next assume that g = [goj/^g/] -1 £ A*. Thus, 

[*]%/] _1 = \qi\ai\pi]- 1 ■ ■ ■ [q m ]a m [p m ]- 1 

in G, where gi,pi, . . . , q m ,Pm £ Q, Oi, • ■ • , a m £ Z 1 U {e}, and (g i; a,,pi) £ 6 for 
1 < i < m. Every A-cycle that occurs in YliLi fe] a «[Pi] _1 can be reduced in G 
to an element from some subgroup of the form X(q,s/). Moreover, since X is 
central in K, we can move each of these elements to the beginning and then to 
the left-hand side {qo]h{qf}~ 1 by taking inverses. Performing this reduction until 
no further A-cycles remain, we obtain in G an equality 

[q^h^f]- 1 = [n 1 ]6 1 [r 1 ]- 1 H6 2 [r 2 ]- 1 • • • [n^/fa] -1 (5) 

where (rij, bi, r^) £ 5 and x £ X{srf)* . Since the right-hand side of (5) is A-cycle 
free, Claim 3 implies x~ x h £ n(L(&/)), i.e., h £ ir{X{srf)* L{srf)). □ 

Let us consider again a finite automaton srf = (Q,£,5,qo, F) over the alpha- 
bet S. A subset P C Q is called admissible if go £ -P and Pflf ^8. For every 
admissible subset PCQwe define the automaton as follows: 

4 = (Px{ii|Ecp, go e r}, s, 8 P , (g , {go}), {(g, P) I g e P n i* 1 }), 
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where the transition relation dp is given by 

S P = {((p, R),a, (q, R U {<?})) | p, 9 eP, (p, a, g) e <5}. 

The automaton sip works as follows. The construction is such that there is a 
run in s/p of a word w from (go, {go}) to (p, i?) if and only if there is a run from 
go top in s/ that visits precisely the states in R. In particular, L(s/p) consists of 
all words w G L(s/) that label an accepting path in srf using exactly the vertices 
from P. 

Lemma 8. For every finite automaton si = (Q, E, 6, qo, F) and h G N, we 
have: h G n(L(s/)) 3P C Q admissible such that h G w(L(sip)). 

Proof. Since L(s/p) C L(,e/) for all admissible subsets, to prove the lemma it 
sumccs to observe that if h — n(w) with w G L(s/), then the set P of states 
visited by a successful run of w is admissible and w G L(s/p). □ 

We are now ready to prove Lemma 6. 

Proof of Lemma 6 Assume that G = *i x H, where X is a proper finite 
normal subgroup of H, has a decidable submonoid membership problem. We 
shall exhibit an algorithm for the rational subset membership problem for H. 
If H is finite, there is nothing to prove. So assume that H is infinite. Let N be 
the centralizer of X in H and let K = (N, t) < G. Then N has finite index in 
H and so it suffices to show that N has a decidable rational subset membership 
problem by [8]. For this, let si = (Q,S,S,qo,F) be a finite automaton and let 
h G N. We show how to decide h G L(s/). By Lemma 8 it suffices to decide 
whether h G tt(L(s/p)) for some admissible subset P C Q. Let us abbreviate the 
automaton s/p by SS. Recall that the set of states of SB is P x {R C P | g G i?}. 

Now we apply Lemma 7 and construct an element g G K and a finite subset 
ACK such that 

/i e n(X(SB)*L(SB)) <^> 9 64*. 

Hence it suffices to prove 7r(L(^)) = n(X(SB)* L(SB)). 

It is immediate that n(L(SB)) C 7r(X(^)*L(^)). So it remains to establish 
that 7r(X(^ > )*L(^ > )) C tt(L(^)). To do this, it suffices to show that 

n(xL(SB)) C tt(L(^)) 

for every x G X(SB) (then we can conclude by induction over the length of the 
word from X(SB)*). Let us take x G X((p,R),SB) for some state (p, R) of SB and 
consider a word xa\ai ■ ■ ■ a m with a\02 ■ ■ ■ a m G L(SB). Hence, there exist states 
(g , Ro), (<7m, R m ) of ^ such that: 

- Rq = {g }, 

- ((gj, ai+i, (gi+i, is a transition of ^ for < i < m — 1, and 

- R m = P, q m G P n F. 
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Note that we have P — {go, • ■ • , q m }- Hence, there exists an j such that qj = p 
(recall that p G P was such that x G X ((p, R) , SB) ) . Since X((p,R),.3$) <Z N C]X 
is central in iV, we have 

xaia2 • • • a m = ai • • • ajxaj + \ ■ ■ ■ a m 

in N. Then there is a loop at state (p, R) = (qj , R) labeled with ci • • • Ck and 
x = 7r(ci---Cfe) by definition of X((p, R),SB). In particular, C\---Ck reads a 
loop in srf at p visiting only states contained in R C P. Now, starting from 
the initial state (qo, Ro) we can read the word a\ ■ ■ ■ ajC\ ■ ■ ■ CfcOtj+i • • • a m on the 
automaton SB. The state reached after reading oq ■ ■ ■ ajC\ ■ ■ ■ Cfcaj + i ■ • • a?, for 
t > J; is °f the form (qi, Si), where Re C S^. Hence, 5 m = P. This shows that 
a\ ■ ■ ■ djCi ■ ■ ■ CfeOj+i • • • a m G L(S§). This completes the proof that N has decid- 
able rational subset membership problem. Therefore, G has decidable rational 
subset membership problem. 

On the other hand, if H has a fixed rational subset with undecidable mem- 
bership problem, then it follows from the construction in [8] that N has a fixed 
rational subset n(L(^)) with undecidable membership problem. It follows from 
Lemma 8 that L(s$p) has an undecidable membership problem for some P. 
Consequently, Lemma 7 provides a fixed finitely generated submonoid of G with 
undecidable membership problem. 

This completes the proof of Lemma 6, thereby establishing Theorem 1. □ 



4 Concluding remarks and open problems 

We have shown that, for every group with at least two ends, the rational subset 
membership problem and the submonoid membership problem are recursively 
equivalent. Moreover, in a previous paper we proved that these two problems 
are recursively equivalent for graph groups as well [13]. 

It is easy to see that there exists a group G with infinitely many ends for 
which submonoid membership is undecidable but the generalized word problem 
(i.e., membership in finitely generated subgroups) is decidable. Take any group H 
for which the generalized word problem is decidable but submonoid membership 
is decidable (e.g., the free metabelian group of rank 2 [14] or the graph group 
defined by a path with 4 nodes [13]). Then the same is true for H * Z, which has 
infinitely many ends. 

The obvious remaining open question is whether there exists a (necessarily 
one-ended) finitely generated group for which rational subset membership is 
undecidable but submonoid membership is decidable. We conjecture that such a 
group exists. We also conjecture that decidability of submonoid membership is 
not preserved by free products, because otherwise rational subset membership 
and submonoid membership would be equivalent for all finitely generated groups. 
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